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PROBLEM TO BE SOLVED; To control an object in a virtual space 
created on a computer through more realistic operation by 
extracting information regarding the position, movement and 
direction of an object to be recognized from its picked-up image 
and controlling the controlled object in the virtual space 
corresponding to the recognized object according to the 
information. 

SOLUTION: An image capture part 1 captures a distance image by 
using a light emission part and a light reception part. Then an image 
process part 4 extracts outline information from the distance image 
data obtained by the image capture part 1 according to the 
reflection coefficient and distance value of a playing tool as the 
recognition object. Then the gravity center position, direction, and 
the moving speed of the gravity center position are calculated from 
the extracted outline information. A virtual space control part 3 
makes the gravity center position, the direction, and the moving 
speed of the gravity center position, extracted from the distance 
image, correspond to the gravity center position, the direction, and 
the moving speed of the gravity center position of a corresponding 
CG model in the virtual space. For the correspondence, the 
coordinates, units and sizes are so converted to match them with 
the inside of the virtual space. 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect 
the original precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim l] The virtual body control approach characterized by extracting the information 
about the location for [ this ] recognition, a motion, and a direction from the image with 
which the candidate for recognition was picturized, and controlling the controlled 
system in the virtual space corresponding to this candidate for recognition based on the 
extracted information. 

[Claim 2] Said image is the virtual body control approach according to claim 1 
characterized by being the depth map which showed the spatial intensity distribution of 
the reflected light from a body. 

[Claim 3] The virtual body control unit characterized by to provide an image -processing 
means perform the image processing for extracting the information about the location 
for [ this ] recognition, a motion, and a direction from the image acquired an image with 
an image acquisition means acquire the image for recognition, and this image 
acquisition means, and the control means which control the controlled system in the 
virtual space corresponding to said candidate for recognition based on the information 
which extracted with this image-processing means. 

[Claim 4] Said image is a virtual body control unit according to claim 3 characterized by 
being the depth map which showed the spatial intensity distribution of the reflected 
light from a body. 

[Claim 5] The record medium which recorded the program which performs the location 
for [ this ] recognition, a motion, an image -processing means to make the information 
about a direction extract, and the control means that makes the controlled system in the 
virtual space corresponding to said candidate for recognition control based on the 
information extracted with this image processing means from the image with which the 
candidate for recognition was picturized and in which machine reading is possible. 



2000-20193 

Translated by Computer 
2 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the virtual body control approach 
which controls a motion of CG model in a virtual space (virtual body), a configuration, 
etc. based on the information on the motion for recognition, a location, etc. extracted 
from the depth map. 

[0002] Moreover, this invention relates to the virtual body control unit which controls a 
motion of CG model in a virtual space (virtual body), a configuration, etc. based on the 
information on the motion for recognition, a location, etc. extracted from the depth map. 
[0003] 

[Description of the Prior Art] Conventionally, with ** implement equipments, such as an 
arcade game machine, many boxing games and games using a motion of human beings, 
such as a shooting game, have been developed. In the boxing game, the pressure when a 
user actually hits a punching bag is measured physically. At this time, only the pressure 
of the moment a user's punch hit the pimching bag is measured. The information in 
what kind of path punch let out with what speed from the user is not used. For this 
reason, it is the monotonous game which competes only for the strength of simple punch. 
[0004] In a shooting game, an emitter is attached to the point of a toy gun and the light 
of the emitter detects where [ on a screen ] it hits. When the trigger of a gun is flipped, it 
judges with the bullet having hit the body upon which hght has shone. Thus, since it is 
simple, it is only the game which competes only for the reflexes which the shown target 
is found quickly [ how ] and are shot with him rather than it says that it competes for 
the arm of shooting. 

[0005] On the other hand, the image picturized using the CCD camera etc. is analyzed, 
and the method of recognizing the location of human being's body, a motion, etc. is also 
taken. By this approach, a travelling direction is first photoed using image pick-up 
equipments, such as a CCD camera. In the case where human being is extracted from 
the image, since the color of a face or a hand is beige, excessive parts, such as the other 
backgrovmd, are removed and pretreatment of starting only an object recognizing [ body 
/ used as an obstruction ] is performed, for example. And the config\iration of an object, a 
motion, etc. are recognized from the image after the processing. 

[0006] First, a pretreatment part called logging for [ this ] recognition is explained. By 
the conventional technique, logging of an object was performed by making an object and 
a certain difference of the other part into a key as an activity means which starts only 
the part of an object to acquire from the image photoed with the camera. 
[0007] As this key, the approach of using change of a hue, the method of using a 
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subtraction image, etc. are used. When starting using a color, the big part of a hue 
difference is extracted, thinning etc. is processed, and an edge is extracted. When 
targeted at human being, paying attention to the flesh color of the parts of a face or a 
hand, it is going to extract only the hue part. The approach using ** and a hue 
difference is difficult for changing, even if it says that it is beige, and there being 
problems, like if the hue of a background is near beige, it is difficult to identify that 
human being is beige, and starting regularly with a color, an include angle, etc. of 
lighting. Moreover, since an image pick-up image all becomes dark in the condition that 
there is no lighting, it is difficult to distinguish a body from the photograph which 
hximan being also took in darkness, 

[0008] When there are many bodies which are moving, the number of motion vectors 
increases suddenly and the increase of the load of calcidation of a motion vector and 
calculation stop moreover; catching up by inter-frame, although it is satisfactory while 
there are few bodies which are moving in this case, although there is also a method 
which computes the inter-frame motion vector of a video image, and analyzes the body 
which is moving. 

[0009] In the game for homes, since presence is increased, a joy stick with the function 
of force feedback and the input device called the oscillating pack which the oscillating 
function attached are beginning to be used. 

[0010] The configxiration of a joy stick is similar to the lever which controls an airplane 
etc. For this reason, surely there is admiration presence in the game which simulates 
operation of the airplane called a flight simulator. In ** and the other actuation, there is 
a trouble used as different user-friendliness from actual actuation. 

[OOll] Similarly, it gets used, and even if an oscillating pack does not have a handle, 
either, it can perform actuation of the car race game which simulated driving of a 
vehicle. With ** and actual actuation, there is a problem that a feeling of actuation is 
greatly far apart, 

[0012] for example, a child - toys, such as a robot and an airplane, using, and a robot 
making the town of blocks or breaking **** as using what exists actually, various 
stories are made and it plays. [ making airplanes fight ] What such a thing that exists 
actually, and the environment of the imagination made in the computer are united, and 
plays was not completed conventionally. 
[0013] 

[Problem(s) to be Solved by the Invention] Thus, special wear or a special instrument 
were not able to be conventionally attached to the body, and the body in the virtual 
space where using special equipments, such as a joy stick, only exists in a computer was 
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not able to be controlled. That is, the thing and virtual space which exist actually were 
not able to be united without sense of incongruity. 

[0014] Then, this invention aims at offering the virtual body control approach which can 
control by more realistic actuation the body in the virtual space made in the computer. 
Moreover, this invention aims at offering the virtual body control unit which can control 
the body in the virtual space made in the computer by more realistic operating 
instructions. 
[0015] 

[Means for Solving the Problem] The virtual body control approach of this invention can 
control by more realistic actuation the body in the virtual space made in the computer 
by extracting the information about the location for [ this ] recognition, a motion, and a 
direction from the image with which the candidate for recognition was pictin-ized, and 
controlling the controlled system in the virtual space corresponding to this candidate for 
recognition based on the extracted information. 

[0016] An image acquisition means by which the virtual body control unit of this 
invention acquires the image for recognition, An image -processing means to perform the 
image processing for extracting the information about the location for recognition, a 
motion, and a direction from the image acquired with this image acquisition means, By 
having provided the control means which controls the controlled system in the virtual 
space corresponding to said candidate for recognition based on the information 
extracted with this image -processing means, the body in the virtual space made in the 
computer is controllable by more realistic actuation. 
[0017] 

[Embodiment of the Invention] Hereafter, the operation gestalt of this invention is 
explained with reference to a drawing. 

(1st operation gestalt) Drawing 1 shows roughly the configuration of the virtual body 
control unit concerning the 1st operation gestalt. 

[0018] As shown in drawing 1 , the virtual body control unit concerning this operation 
gestalt The depth map acquisition section 1 which receives the reflected light, and 
acquires a depth map, for example, is indicated by Japanese Patent Application No. No. 
299648 [ nine to ], The image -processing section 4 which extracts the information which 
analyzes the acquired depth map and is made into the purpose about the candidates for 
recognition concerned, such as a profile for recognition, the extract of a center of gravity 
and calculation of the distance to an object, passing speed for recognition, and 
calculation of a movement vector, The three dimension or two-dimensional virtual space 
created by CG (computer graphics), The shape memory section 2 which memorizes the 
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configxiration of the three dimension shown in the virtual space, or a two-dimensional 
virtual body (for example, it is hereafter called CG model with a person, a ** implement, 
etc.), the arrangement location of a virtual space, etc., The virtual space control section 
3 which controls the configuration of CG model memorized by the shape memory section 
2, a motion, etc. based on the analysis result of the image-processing section 4, For 
example, it consists of displays, such as a large- sized liquid crystal panel, and consists 
of the presentation sections 5 which visuahze the virtual space as a three dimension or 
a two-dimensional CG image, and visualize the configuration of the basis of control of 
the virtual space control section 3, and CG model as a controlled system memorized by 
the shape memory section 2, a motion, etc. 

[0019] Here, the depth map acquired in the image acquisition section 1 and the image 
acquisition section 1 is explained briefly. The appearance of the image acquisition 
section 1 is shown in drawing 2 . The Hght sensing portion 103 constituted from an area 
sensor (not shown) at a circular lens and its posterior part by the center section is 
arranged, and two or more (for example, six pieces) light-emitting parts 101 which 
consist of LED which irradiates light, such as infrared radiation, are arranged at equal 
intervals along with the profile around the circular lens. 

[0020] It is reflected by the body, and is condensed with the lens of a hght sensing 
portion 103, and the light irradiated from the light- emitting part 101 is received by the 
area sensor at the posterior part of a lens. An area sensor is the sensor arranged in the 
shape of [ of 256x256 ] a matrix, and the reinforcement of the reflected light received by 
each sensor in a matrix serves as a pixel value, respectively. Thus, the acquired image is 
a depth map as intensity distribution of the reflected light as shown in drawing 4 . 
[0021] Drawing 3 is what showed the example of a configuration of the image 
acqmsition section 1, and mainly consists of a light-emitting part 102, a light sensing 
portion 103, the reflected light extract section 102, and the timing signal generation 
section 104. A light-emitting part 101 emits light in the light which carries out 
on-the-strength fluctuation in time according to the timing signal generated in the 
timing signal generation section 104. This light is irradiated by the object object which 
is ahead [ hght emitting part]. 

[0022] A light sensing portion 103 detects the amount of the reflected hght with the 
object object of the hght which the light-emitting part 101 emitted. The reflected light 
extract section 102 extracts the spatial intensity distribution of the reflected hght 
received by the light sensing portion 103. Since the spatial intensity distribution of this 
reflected light can be regarded as an image, this is hereafter called a depth map. 
[0023] Not only the reflected light by the object of the Hght generally emitted from a 
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light-emitting part 101 but outdoor daylight, such as illumination hght and sunlight, 
receives a light sensing portion 103 to coincidence. Then, the reflected light extract 
section 102 takes out only a reflected Ught component with the object object of the light 
from a hght emitting part 101 by taking the difference of the amount of the light which 
received Ught while the light- emitting part 101 was emitting light, and the amount of 
the hght which received hght while the light-emitting part 101 was not emitting hght. 
[0024] In the reflected light extract section 102, the intensity distribution, i.e., a depth 
map as shown in drawing 4 , are extracted from the reflected light received by the light 
sensing portion 103. By drawing 4 , since it is easy, the case of the 8x8-pixel depth map 
which is a part of 256x256-pixel depth map is shown. 

[0025] The reflected light from a body decreases sharply as an objective distance 
becomes large. When objective front faces are uniformly scattered about in light, the 
light income per 1 pixel of depth maps becomes small in inverse proportion to the 
square of the distance to a body. 

[0026] In drawing 4 , the value (pixel value) of the eel under matrix shows the strength 
of the acquired reflected hght with 256 gradation (8 bits). For example, there is a eel in 
which the eel with the value of "255" has the condition which approached the depth map 
acquisition section 1 most, and the value of "0" in the distance from the depth map 
acquisition section 1, and it is shown that the reflected hght does not reach even the 
depth map acquisition section 1. 

[0027] Each pixel value of a depth map expresses the amount of the reflected Ught 
which received light by the tmit Ught sensing portion corresponding to the pixel. 
Although the reflected light is influenced by objective properties Gight is absorbed 
[ carrying out specular reflection, being scattered about, ]), the direction of objective, 
objective distance, etc., when the whole body is a body uniformly scattered about in light, 
the amount of reflected lights has close relation with the distance to a body. Since a 
hand etc. has such a property, the depth map at the time of holding out a hand ahead of 
the image acquisition section 1 can acquire a three-dimension-image as shown in 
drawing 5 reflecting the distance to a hand, the inclination (distance differs partially) of 
a hand, etc. 

[0028] In the image-processing section 4, the various image processings for extracting 
the information on the configuration of image pick-up objects, such as edge logging 
(profile extract of an image pick-up body), a center-of-gravity extract, area calculation, 
calculation of the distance to an image pick-up body, and calculation of a motion vector, 
a motion, a location, etc, from depth map data as shown in drawing 4 sent firom the 
image acquisition section 1 are performed. 
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[0029] The image acquisition section 1 picturizes the condition that the user is playing 
the game with the sword 21 as a ** implement, as shown in drawing 7 . The image 
acquisition section 1 acquires the ** implement as a candidate for recognition, i.e., the 
depth map of a sword, (refer to drawing 11 (a)), and the image-processing section 4 
analyzes the depth map. 

[0030] In the image-processing section 4, the three dimension-location of the profile of a 
sword, a center of gravity, and a center of gravity, a direction, a rate, etc. are extracted 
from a depth map, and they are passed to the virtual space control section 3. The virtual 
space created by the three dimension or two-dimensional CG (computer graphics) 
memorized by the shape memory section 2 as shown in drawing 8 , for example is 
displayed, the configurations of a user's CG model 23 and the CG model 22 of a sword 
memorized by the shape memory section 2, a motion, etc. are controlled by the virtual 
space control section 3 based on the information extracted in the image-processing 
section 4 there, and it shows real time at the presentation section 5. Thereby, a user can 
play a game with feeling in which he entered the virtual space. 

[0031] In the image -processing section 4, the profile information for [ which is in the 
depth map concerned as shown in drawing 6 first ] recognition is extracted from the 
reinforcement of the reflected hght by the difference in the reflection coefficient of an 
image pick-up body, and physical relationship with the other image pick-up body for 
recognition. 

[0032] For example, if the image acquisition section 1 stations the user who gained the 
sword in the location which can be picturized from a transverse plane, in a depth map 
as shown in drawing 11 (a) acquired in the image acquisition section 1, as for the ** 
implement as a candidate for recognition, i.e., a sword, distance with the image 
acquisition section 1 will exist most in near from other image pick up objects. That is, 
since the reflected light which the image acquisition section 1 acquires becomes strong 
so that distance with the image acquisition section 1 is short, a pixel value becomes 
large (if a pixel value is large, it will become more close to black). The part 31 of the 
point of the sword of the nearest part is the blackest, and a pixel value becomes small 
gradually with the part 32 near the shank of a sword, and the hand 33 of a user with a 
sword (becoming white), and since a background is the furthest, a pixel value is mostly 
set to "0." 

[0033] The distance value from the image acquisition section 1 to an image pick-up body 
can be measured such to not only 4 gradation but to 256 gradation or twist high degree 
of accuracy. Calculation of the distance d to the body with which the profile was 
extracted is first asked for the representation pixel value of the image near the 
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center-of-gravity location of the body concerned. As a representation pixel value, 
although there are the average, a recently side value, etc. partly, suppose that a recently 
side value is used here. The strength of the reflected light from the body concerned 
becomes small in inverse proportion to the square of the distance to a body. That is, 
when a pixel value is set to P (i, j), the relation of the pixel value and distance value w is 
P(i, j)=K/w2. (1) 

It can express. Here, K is defined for every image pick-up body, is the multiplier 
adjusted so that the value of the pixel value P of the image pick-up body concerned (i, j) 
might be set to "255" at the time of w- 0.5m, and also calls it a reflection coefficient. The 
distance value w can be calculated by solving a formula (l) about w. 
[0034] Supposing the reflection coefficient of the sword for recognition is larger than the 
thing of other image pick-up objects (for example, magnitude of extent to which the 
pixel value becomes larger than the pixel value of the image of a hand even when it is 
more back than the hand which grasps a sword). And it masks with a distance value to a 
depth map. Supposing it extracts only the image field of large (that is, near part) 
gradation of a value most among 4 gradation now, in the case of drawing 11 (a), only the 
part 31 of the point of a sword can be started. This hits setting to the value (for example, 
"0") except the pixel of the value more than a certain constant value with the lowest 
pixel value. 

[0035] What is necessary is to compare the pixel value of a ****** pixel, to put a 
constant value only into the place whose pixel value is more than the constant value 
alpha, and just to extract the pixel of the continuous image field where the same 
constant value was assigned, in order to extract profile information. 
[0036] Namely, if the pixel value in the coordinate location on the matrix of drawing 4 (i, 
j) is set to P (i, j) and the pixel value of profile information is set to Q (i, j) - P (i, j) * P 
(i-1, j) "} " > -■ alpha - and - {- P G, j) ■ P (i, j-l) ") " > " alpha and - {" P (i, j) - P (i+1, 
j) ••} " > " alpha " and - { - P (i, j) ■ P G, j+l) ") " > ■ alpha - the time Q(i, j) =255- 
The profile information on an image pick-up body like drawing 6 can be acquired by 
being referred to as Q(i, j) = 0 at the times other than the above. 

[0037] Thus, the profile information on the part 31 of the point of a sword as shown in 
drawing 11 (b) from drawing 11 (a) can be started. The image processing section 4 
calculates the center of gravity Rg of the part 31 of the point of the sword concerned 
from the profile information on the part 31 of the point of a sword. 

[0038] IMoreover, as shown in drawing 11 (b), the direction Dg of a sword is computable 
by asking for the normal vector of a field including a center-of-gravity location. A normal 
vector is a perpendicular straight line to the tangential plane to the field which passes 
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along a center of gravity Rg and includes a center of gravity Rg. 

[0039] It can judge whether it is extracted from the center -of- gravity location of the 
sword which is a candidate for recognition, and the last frame, the center -of* gravity 
location of the sword which is a candidate for recognition is compared, and the body is 
moving. When it judges that it is moving, the motion vector showing the variation and 
the change direction of a center of gravity is computed. 

[0040] When the center-of-gravity location of one frame ago is made into Rg', for the 
variation of a center-of*gravity location, the rate Vg by which a center of gravity Rg 
moves will be Vg=(Rg"Rg') x30, if it becomes (Rg-Rg') and the depth map of 30 frames is 
acquired in 1 second. (2) 
since " it can ask. 

[0041] Thus, information, such as distance from the depth map acquired in the image 
acquisition section 1 to the center of gravity the configuration for recognition, a 
direction, and for recognition and a center of gravity and a motion, can be extracted. 
Next, with reference to the flow chart shown in drawing 9 , processing actuation of the 
virtual body control unit of drawing 1 is explained. 

[0042] First, if started with the injection of a power source, or initiation directions of 
operation, the virtual space control section 3 will display the virtual space of the three 
dimension memorized by the shape memory section 2 or two-dimensional CG on the 
presentation section 5. And it shows by making a rate v into initial value in the virtual 
space currently displayed on the presentation section 5, making as the center-of-gravity 
location r and Direction d a person's CG model (here, it corresponds to a user) 
memorized by the shape memory section 2. Moreover, the value of the center-of-gravity 
location R, Direction D, and a rate V is made into initial value, and CG model Qiere, it 
corresponds to the sword which the user has gained) of a ** implement is shown in the 
virtual space currently displayed on the presentation section 5 (step Si). 
[0043] Then, the image acquisition section 1 acquires a depth map at the rate of about 
30 sheets in 1 second, using a light- emitting part 101 and a light sensing portion 103 as 
shown in drawing 2 (step S2). Generally, interactive processing with a user is performed, 
i.e., it is possible for the upper limit of the speed of response which a user can recognize 
that the other party, such as a computer, has reacted to their action to be 0.2 seconds, to 
process on real time enough, if per second 30-sheet image acquisition can be performed, 
and to return a natural response for a user. 

[0044] Next, the image -processing section 4 is based on the reflection coefficient and 
distance value of the ** implement which is a candidate for recognition as mentioned 
above, i.e., a sword, from the depth map acquired in the image acquisition section 1, for 
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example, depth map data as shown in drawing 4 , and is a sword (the profile 
information on the part (refer to drawing 11 (a)) 31 of the point of a sword is extracted in 
detail (step S3).). And the center-of gravity location Rg, the direction Dg of a sword, and 
the rate Vg of a motion of a center-of-gravity location are computed from the profile 
information on this extracted sword (step S4). 

[0045] In the virtual space control section 3, the center-of-gravity location Rg of the 
sword extracted from this depth map, the direction Dg of a sword, and the rate Vg of a 
motion of a center-of-gravity location are matched with the center-of-gravity location R 
of CG model corresponding to the sword concerned in a virtual space, Direction D, and 
the rate V of a motion of a center of-gravity location. This matching changes a 
coordinate, a unit, and magnitude so that the center-of-gravity location R in the actual 
space extracted from the depth map, Direction D, and the rate V of a motion of a 
center-of-gravity location may be suited in a virtual space (step S5). 
[0046] Furthermore, the virtual space control section 3 computes the center-of-gravity 
location r of a user's CG model, Direction d, and the rate v of a motion of a 
center-of-gravity location from the center-of-gravity location R of CG model of a sword, 
Direction D, and the rate V of a motion of a center of-gravity location (step S6). 
[0047] For improvement in the speed, count is simplified and the center-of-gravity 
location of a user's CG model is computed by the following technique. Drawing 12 is 
drawing for explaining how computing the center-of gravity location P of CG model of a 
** implement (sword), the center-of-gravity location p of Direction D to a person's (user) 
CG model, and Direction d. the virtual space control section 3 -- for example, drawing 13 
memorized by predetermined memory -■ ** with reference to the table which 
memorized calculation criteria [ like ], the center-of-gravity location P of CG model of a 
sword, the center-of gravity location p of Direction D to a user's CG model, and 
Direction d are computed. 

[0048] xz flat surface where the direction D of CG model of a sword includes the center 
of gravity R of CG model of a sword, and the angle to make are set to theta, and yz flat 
surface where the direction D of a sword includes the center of gravity R of CG model of 
a sword, and the angle to make are set to phi. Here, the time of the direction D of the 
normal vector of a sword being located on right-hand side across yz flat surface in phi is 
made forward, and the time of being in left-hand side is defined as negative and a 
conceptual target. 

[0049] The time smaller than the value alpha which has theta on the calculation criteria 
shown in drawing 13 , and theta are with the larger time than alpha (namely, when the 
sword itself is near horizontally), (namely, when most sword itself is a perpendicular) 
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the time (phi is negative) of the presumed approach roughly being divided into two and 
the time (phi being forward) of a sword being in right-hand side across yz flat surface 
about each further and a sword being in left-hand side across yz flat surface ■- a case 
dividing *• carrying out " judging - coming - **** . 

[0050] At the perpendicularly near time, a user is in the condition which bends a wrist, 
brings close to the body and has a sword, and a sword is in the condition which a user 
lengthens a wrist, and a sword separates from the body at the horizontally near time, 
and has a sword. 

[0051] On the calculation criteria shown in drawing 13 , the center -of -gravity location r 
and Direction (the direction of a center of gravity) d of CG model of a user are computed 
as follows from the center of gravity R and Direction D of CG model of a sword by 
presuming a gestalt when a user's CG model gains the sword concerned from the 
direction D of CG model of a sword. 

[0052] A sword is almost perpendicular, and when having turned to the right, the 
center-of-gravity location r of a user's CG model (x3, y3, z3) is set to X3=xl-beta, 
y3=yl*gamma, and Z3=zH-delta to the center of gravity location R of CG model of a 
sword (xl, yl, zl) (when phi is forward), beta, gamma, and delta are constants, beta is a 
value proportional to the magnitude of the hand a user's CG model, gamma is 
magnitude which is mostly equivalent to the die length of CG model of a sword, and 
delta is the magnitude equivalent to one half of the load arms of a user's CG model. 
[0053] A sword is almost perpendicular, it is adding the center-of-gravity location r of a 
user's CG model in x components instead of lengthening beta, when having turned to 
the left (when phi's being negative), and right and left are reversed. That is, it is set to 
X3=xl+beta, y3=yl-gamma, and Z3=zl+delta. 

[0054] Since the direction d of a user's CG model is considered that the body is also 
mostly concurrent with the sword when a sword is almost perpendicular, the direction D 
of a sword (x2, y2, z2) is used as it is as a direction d of a user's CG model (X4, Y4, Z4). 
[0055] When X4=x2, y4=y2, and Z4=z2 sword is almost level and has turned to the right, 
(when phi is forward) The center of- gravity location r of a user's CG model (x3, y3, z3) is 
set to X3=xl*epsilon, Y3=yl-zeta, and Z3=zl+eta to the center-of-gravity location R of a 
sword (xl, yl, zl). epsilon, zeta, and eta are constants, and epsilon is a value 
proportional to the magnitude of the hand a user's CG model, and, moreover, is a larger 
value than beta, zeta becomes [ gamma ] with the value equivalent to the width of face 
of a sword, and is a small value, eta is the magnitude equivalent to three fourths of the 
load arms of a user's CG model. 

[0056] A sword is almost level, and when having turned to the left, the center-of-gravity 
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location r of a user's CG model is added instead of lengthening epsilon in x components 
(when phi is negative). That is, it is set to X3=xl+epsilon, y3=yl-2eta, and Z3=2l+eta. 
[0057] When a sword is almost level, since it thinks [ having tended to have rotated 
about 90 degrees in the vertical plane, and ] to a sword, the direction d of a user's CG 
model rotates the direction of a sword 90 degrees, and uses the body as it is as a 
direction d of a user's CG model (X4, Y4, Z4). That is, the rate v of a motion of X4=x2, 
y4=y2, and Z4=y2 user's CG model may be beforehand defined corresponding to the rate 
V of a motion of CG model of a ** implement. 

[0058] As shown in drawing 14 , the configuration of a person's CG model, CG model of a 
** implement, and the other characters in a virtual space etc. is memorized by the 
shape memory section 2. In drawing 14 , data, such as a configuration of the actual 
three-dimension CG model of an object called a person's CG model, exist after the 
memory address shown with the pointer "Pointer 1." Moreover, the bounding box shows 
six top-most-vertices coordinates of the rectangular parallelepiped circumscribed to a 
three -dimension CG model. With a bounding box, it is confirmed whether the bodies in a 
virtual space have collided (interference check). If either of six top-most vertices of this 
bounding box is contained in a partner's bounding box, it considers that it has interfered. 
[0059] In step S6, after searching for the center-of-gravity location R of CG model of a 
sword, the center -of- gravity location r of Direction D to a user's CG model, and Direction 
d, the virtual space control section 8 reads the configuration data of a user's CG model 
from the shape memory section 2. And the center-of-gravity location of this 
configuration data is shown in the virtual space already shown to the presentation 
section 5 so that r and its direction may serve as the rate v of d and a motion (step S7). 
An example of the display screen of the presentation section 5 at this time is shown in 
drawing 8 . 

[0060] The virtual space control section 3 performs the interference check of a user's CG 
model and other bodies all over a virtual space by comparing the bounding box 
memorized by the shape memory section as shown in drawing 14 at the same time it 
presents a user's CG model (step S8). If it has interfered, since a user's CG model will 
have colhded with other bodies, impulsive sound is generated from the presentation 
section 5 (step Sl5 of drawing 10 ). As for impulsive sound, it is also possible to also 
decide a specific sound or for there to be a different sound for every body. Moreover, a 
user's CG model is rebounded by having collided by the body which collided. According 
to the location which interfered in the amount of rebound phenomenon and the direction 
of the rebound phenomenon of [ when interfering with a user's CG model for every 
various characters which exist in a virtual space ], it sets beforehand, it computes how 
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much it is rebounded from the center-of-gravity location r and Direction d of the value 
and the present user's CG model, and the center-of-gravity location p and Direction d of 
CG model of a user are updated (step S16). 

[0061] Since the center-of*gravity location p and Direction d of CG model of a user were 
updated, the center-of-gravity location P and Direction D of CG model of a sword are 
re(step S17) calculated again. After performing processing of step Sl5 - step Sl7, an 
interference check is further performed to step S7 at return and step S8, and when a 
user's CG model has interfered with the body when it is judged that it has not 
interfered with the body in other virtual spaces, it progresses to step S9 of drawing 10 . 
[0062] In step S9, the virtual space control section 3 reads the configuration data of CG 
model of a sword from the shape memory section 2. And the center -of gravity location of 
this configuration data is shown in the virtual space already shown to the presentation 
section 5 so tliat R and its direction may serve as D and the rate V of a motion (step S9). 
[0063] The virtual space control section 3 performs the interference check of CG model 
of a sword, and other bodies all over a virtual space by comparing the bounding box 
memorized by the shape memory section as shown in drawing 14 at the same time it 
presents CG model of a sword (step SlO). If it has interfered, since CG model of a sword 
will have collided with other bodies, the explosive sound according to a rate V is 
generated from the presentation section 5, for example (step Sll). And for example, a 
body is hard, since the center-of gravity location and direction of a sword change when a 
sword bounds and is returned, the calculation is performed (step Sl2), and CG model of 
a sword is again re(step Sl3) displayed based on the result. Here, processing will be 
ended if there are directions of termination (step S14). 

[0064] As explained above, according to the operation gestalt of the above 1st, the 
configuration for [ this ] recognition and a motion are extracted from the depth map by 
which the candidate for recognition (for example, sword as a ** implement which the 
user has in the hand) acquired in the image acquisition section 1 was picturized in the 
image -processing section 4. Based on the configuration and motion which were 
extracted, and the controlled system in the virtual space corresponding to this 
candidate for recognition (CG model of a user or a sword) moves. Further By controlling 
a configuration etc., a user can attach special wear and a special instrument to the body, 
or can control by more realistic actuation the body in the virtual space made in the 
computer without obstructive things, such as a cable extended from a body. That is, the 
body which exists in a virtual space is operated with feeling in which he exists all over a 
virtual space, and simulation with a touch of reality can be performed, or it can be 
enjoyed as a game (game). 
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[0065] Moreover, if it carries out from the contractor who installs such an arcade game 
when applying the virtual body control unit concerning the above-mentioned operation 
gestalt to an arcade game Consideration of the useless time amount at the time of a 
user detaching and attaching special wear, since special wear, an instrument, a cable, 
etc. are unnecessary, Since it is released from sanitary consideration of the wear worn 
among two or more users, the consideration to wear of the facihty accompanying a 
motion of a user, etc., the cost concerning employment of a game machine is sharply 
reducible. 

[0066] In addition, with the operation gestalt of the above 1st, although the number of 
the image acquisition section 1 was one, it is not necessarily hmited to this. You may 
make it compute the center of gravity location for [ in a depth map ] recognition, the 
direction of a center of gravity, the rate that moves by installing two or more cameras of 
a configuration as shown in drawing 3 for acquiring a depth map, performing the same 
image processing as the above-mentioned to each of the depth map acquired with each 
camera, and unifying the image -processing result. 

[0067] Moreover, it is also possible by using two or more CCD cameras to extract 
three-dimension-information (the center-ofgravity location for recognition, the 
direction of a center of gravity, rate that moves) from the image for [ in an image ] 
recognition like the above-mentioned operation gestalt. Becoming late as compared with 
the image-processing rate to a depth map cannot deny the image -processing rate in this 
case. 

(2nd operation gestalt) Drawing 15 shows roughly the configuration of the virtual body 
control \xnit concerning the 2nd operation gestalt of this invention. In addition, in 
drawing 15 , the same sign is given to the same part as drawing 1 , and a different part 
is explained. With the 1st operation gestalt, when he moved a hand and the body, a user 
is that CG model of the user concerned and CO model of a sword which are shown in the 
screen follow in footsteps and move, and sensed presence. The sensibihty which collided 
was also felt by the impulsive sound emitted when other bodies in a virtual space are 
collided with, or the explosive sound. 

[0068] With the 2nd operation gestalt, on the other hand, by providiag further the 
feedback generation section 6 and the feedback presentation section 7 When a user's CG 
model and CG model of a sword interfere with other bodies in a virtual space, while 
generating impulsive sound and an explosive sound in the presentation section 5 For 
example, it enables it to feel the presence which has a touch of reality more to a user by 
generating and showing force (force) feedback. 

[0069] When a user's CG model and CG model of a sword interfere with other bodies in a 
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virtual space, the magnitude of the force of force feedback computes the feedback 
generation section 6. 

[0070] The feedback presentation section 7 presents an actual sense of force by making 
a motor generate running torque according to the magnitude of the force fee dock which 
was taught into the ** implement (here sword) which the user has gained, for example, 
was computed in the feedback generation section 6. As a motor, a DC motor is used, for 
example, 

[0071] S is computed whenever [ impact ] from the rate Vg to which weight M of the 
sword as a ** implement which the user has in the hand, and the center of gravity of the 
sword concerned extracted from the image move by the feedback generation section 6. 
[0072] 

It is S=KMV2 whenever [ impact ] . (K is a proportionality constant) 

The output (cvirrent value) of the DC motor which constitutes the feedback presentation 

section 7 embedded at the ** implement based on S whenever [ this impact ] is 

controlled. Force feedback is shown in the feedback presentation section 7 because the 

torque of the opposite direction occurs because a motor rotates suddenly. 

[0073] In addition, in order to transmit control information to the feedback presentation 

section 7 (for example, DC motor) embedded at the ** implement, you may be a wire 

communication and IrDA Qnfrared Data Association) etc. may be radio. 

[0074] since a user has feedback of the impulse force according to a scene while the body 

in the virtual space made in the computer is controllable by more realistic actuation 

according to the operation gestalt of the above 2nd, as explained above -■ more real 

somesthesis - ******** -- things become possible. 

[0075] In addition, in the configuration of drawing 1 and the virtual body control unit of 
drawing 15 , processing actuation of each configuration section except the image 
acquisition section 1 can also be stored and distributed to record media, such as a floppy 
disk, a hard disk, CD-ROM, DVD, and semiconductor memory, as a program which a 
computer can be made to execute. 

[0076] Moreover, although the above 1st and the 2nd operation gestalt explained the 
virtual body control unit of this invention taking the case of the case where it applies to 
a game, it cannot be overemphasized that it can apply to the practical system to broad 
industrial fields to which the applicability of this invention is not restricted in this case 
for example, which are not allowed failure, such as education, training, etc. of a 
dangerous activity, medical simulation, etc. 

[0077] furthermore , although the above 1st and the 2nd operation gestalt explained 
taking the case of the case where the candidate for recognition in an image be the ** 
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implement ( sword ) which the user have in the hand , it do not restrict in this case , but 
the candidate for recognition be the user itself and it can perform control the virtual 
body in a virtual space based on information , such as a location extracted from this user 
image , a motion , and a configuration , as well as the above-mentioned . 
[0078] 

[Effect of the Invention] As explained above, according to this invention, the body in the 
virtual space made in the computer is controllable by more realistic actuation. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing l1 Drawing having shown roughly the configuration of the virtual body control 
unit concerning the 1st operation gestalt of this invention. 

[Drawing 2l Drawing having shown an example of the appearance of the depth map 
acquisition section. 

[Drawing 3l Drawing having shown the example of a configuration of the depth map 
acquisition section. 

[Drawing 4] Drawing having shown an example of the depth map which makes 
reinforcement of the reflected light a pixel value. 

[Drawing 5] It is three dimension drawing which expressed about the depth map of a 
matrix type as shown in drawing 4 . 

[Drawing 61 Drawing having shown an example of the profile image of the body 
extracted from the depth map. 

[Drawing 7l Drawing in which the user showed an example of the image pick up 
situation of the image acquisition section which picturizes the condition of playing the 
game with the sword as a ** implement. 

[Drawing 8l Drawing having shown an example of the virtual space displayed on the 
display screen of the presentation section. 

[Drawing 9l The flow chart for explaining processing actuation of the virtual body 
control device of drawing 1 . 

[Drawing lOl The flow chart for explaining processing actuation of the virtual body 
control device of drawing 1 . 

[Drawing 111 It is drawing for the (a) Fig. to be drawing for explaining how to extract 
the image of a sword based on the physical relationship of the sword for recognition, and 
a hand in drawing having shown an example of the depth map acquired by the image 
acquisition section, and for the (b) Fig. explain the normal vector of the sword extracted 
from the depth map. 

[Drawing 12l Drawing for explaining how computing the center-of-gravity location P of 
CO model of **** (sword), the center-of gravity location p of Direction D to a person's 
(user) CO model, and Direction d. 

[Drawing 13l Drawing having shown an example of a table which memorized the 
calculation criteria for computing the center-of-gravity location P of CG model of a 
sword, the center-of-gravity location p of Direction D to a user's CG model, and 
Direction d. 
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[Drawing 14l Drawing having shown the example of a data storage of the virtual body 
in the shape memory section 

[Drawing 15l Drawing having shown roughly the example of a configuration of the 
virtual body control unit concerning the 2nd operation gestalt of this invention. 
[Description of Notations] 

1 " Image acquisition section 

2 - Shape memory section 

3 ■■ Virtual space control section 

4 - Image -processing section 

5 " Presentation section 

6 Feedback generation section 

7 - Feedback presentation section 



